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Abstract 

Background: Current Medical Research Council (MRC) guidance on complex interventions advocates pilot trials and 
feasibility studies as part of a phased approach to the development, testing, and evaluation of healthcare 
interventions. In this paper we discuss the results of a recent feasibility study and pilot trial for a randomized 
controlled trial (RCT) of pelvic floor muscle training for prolapse (ClinicalTrials.gov: NCTOl 136889). The ways in which 
researchers decide to respond to the results of feasibility work may have significant repercussions for both the 
nature and degree of tension between internal and external validity in a definitive trial. 

Methods: We used methodological issues to classify and analyze the problems that arose in the feasibility study. 
Four centers participated with the aim of randomizing 50 women. Women were eligible if they had prolapse of any 
type, of stage I to IV, and had a pessary successfully fitted. Postal questionnaires were administered at baseline, 6 
months, and 7 months post-randomization. After identifying problems arising within the pilot study we then 
sought to locate potential solutions that might minimize the trade-off between a subsequent explanatory versus 
pragmatic trial. 

Results: The feasibility study pointed to significant potential problems in relation to participant recruitment, 
features of the intervention, acceptability of the intervention to participants, and outcome measurement. Finding 
minimal evidence to support our decision-making regarding the transition from feasibility work to a trial, we 
developed a systematic process (A process for Decision-making after Pilot and feasibility Trials (ADePT)) which we 
subsequently used as a guide. The process sought to: 1) encourage the systematic identification and appraisal of 
problems and potential solutions; 2) improve the transparency of decision-making processes; and 3) reveal the 
tensions that exist between pragmatic and explanatory choices. 

Conclusions: We have developed a process that may aid researchers in their attempt to identify the most 
appropriate solutions to problems identified within future pilot and feasibility RCTs. The process includes three key 
steps: a decision about the type of problem, the identification of all solutions (whether addressed within the 
intervention, trial design or clinical context), and a systematic appraisal of these solutions. 
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Background 

Over the past 15 years, there has been increasing formal 
guidance and a growing number of methodological pa- 
pers, opinions, and commentaries published highlighting 
the need for greater attention to be paid to both the 
initial development of healthcare interventions and the 
design of their evaluation [1-3]. Such attention is under- 
standable given both the increasing cost associated with 
delivering high quality trials [4], the number of interven- 
tions that fail to demonstrate effectiveness, and the diffi- 
culties in translating any demonstrable benefits into 
real-world contexts [5]. There have been suggestions 
that many of these translational issues may be ame- 
liorated by addressing transferability to practice issues 
within earlier stages of intervention design [6-8]. 

Common recommendations include the need for thor- 
ough feasibility work to be conducted in order to pre- 
empt problems with the intervention itself, to support 
the implementation of a robust trial or other evaluative 
design, and for transferability to real-world practice. For 
example, current Medical Research Council (MRC) guid- 
ance on complex intervention evaluation advocates two 
distinct processes as part of a phased approach to the 
development, evaluation, and implementation of com- 
plex interventions in healthcare [1,2]. This includes 
feasibility studies (asking questions about whether the 
study can be done) and pilot trials (a miniature version 
of the main trial), which aim to test aspects of study 
design and processes for the implementation of a larger 
main trial in the future [2,9,10]. This increasingly de- 
tailed preparatory stage aims to maximize the likelihood 
of delivering a definitive trial that is high in terms of 
both internal validity (the scientific robustness of the 
trial) and external validity (its generalizability to routine 
real- world contexts). Simultaneously achieving both in- 
ternal and external validity may be problematic; indeed, 
they may be in tension [11,12]. Consequently, two cat- 
egories of trial have entered the methodological language 
[13,14]: explanatory trials, which focus on establishing 
efficacy (that is, can the intervention work?) with result- 
ant high internal but low external validity; and pragmatic 
trials, which focus on effectiveness (that is, does the 
intervention work in real-world contexts?) with resultant 
lower internal but higher external validity. 

Despite the recognized importance of feasibility and 
pilot work, two problems exist. First, several authors have 
suggested that the results of most contemporary pilot 
trials or feasibility studies are published with a focus on 
outcomes [9,15,16] rather than the methodological issues 
which these studies are commonly intended to investigate. 
Such studies may fail to detect or address any of the 
subsequent threats to internal or external validity, and 
certainly do not have sufficient sophistication to identify 
trade-offs or potential solutions. Second, the reporting of 



pilot trials and feasibility studies may be sub-optimum. 
Given their importance, such studies ought to be carefully 
designed, with clear aims and objectives to support their 
own scientific validity in order to appropriately inform 
decisions regarding any future trial [16,17]. Yet in a recent 
review of published pilot trials and feasibility studies only 
56% of the identified studies reported issues that would be 
useful in the development of a main trial [15]. 

In this paper we present an example of a feasibility 
study, which included a pilot trial, of a complex inter- 
vention. We reflect upon the importance of feasibility 
work and how a better understanding of the processes of 
developing the definitive trial following a feasibly study 
or pilot trial might contribute to the debate regarding 
explanatory and pragmatic trials. We discuss the meth- 
odological implications for future feasibility study con- 
duct and reporting. The example relates to a complex 
intervention for the conservative management of female 
pelvic organ prolapse. It was hypothesized that un- 
dertaking pelvic floor muscle training (PFMT) with a 
vaginal pessary in place would lead to additional im- 
provements in prolapse symptoms and quality of life for 
women with prolapse beyond that expected from a 
pessary alone. PFMT and pessaries are commonly used 
independently to treat pelvic organ prolapse [18,19]. 
There is growing level one evidence to support the ef- 
fectiveness of PFMT [20,21]. In addition, some research 
suggests that use of a pessary may improve the anatom- 
ically defined stage of prolapse [22], and a woman's 
symptoms and quality of life [23-25]. However, there is 
no evidence relating to the combination of these treat- 
ments, and whether a combination is more effective for 
prolapse than treatment with either alone. 

Due to the complex nature of PFMT, which involves 
multiple patient/therapist contacts and a progressive 
tailored home exercise programme, there was a need to 
consider evaluation of the PFMT plus pessary interven- 
tion in the light of the MRC s framework for evaluation 
of complex interventions [2]. Furthermore, while there 
were some aspects of design which could be confidently 
determined based on previous research in this clinical 
area (for example, the primary outcome measure had 
been developed and tested [26]), there were other as- 
pects of the design for which evidence was lacking, and 
therefore a feasibility study, including a pilot trial, was 
deemed necessary prior to undertaking a definitive trial. 

The aims of this feasibility study and pilot trial were: 
to determine the feasibility of conducting an RCT of the 
effectiveness of a PFMT intervention in addition to a va- 
ginal pessary versus a vaginal pessary alone in improving 
prolapse symptoms, quality of life, and prolapse severity 
(including assessing issues of eligibility, checking all 
components of the intervention work together); and to 
develop and test the methods for a main trial (including 
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sample size calculations, eligibility, recruitment, consent, 
randomization, adherence to the intervention, partici- 
pant retention, logistics of a multicenter randomized 
controlled trial (RCT)). 

The aims of this paper are: to outline the processes we 
used to explore the best pathway forward from feasibility 
study and pilot trial to main trial and real-world' im- 
plementation; to describe a process designed to assist 
researchers in making best use of the findings from their 
feasibility and pilot work to inform their subsequent 
decisions regarding a follow-on trial; and to illustrate the 
use of the process using an exemplar feasibility study, 
the Pessary Plus Physiotherapy for Pelvic Organ Prolapse 
(PEPPY) study. 

Methods 

PEPPY feasibility study and pilot trial methods 

The research design assessed both issues of feasibility 
and issues of implementation for the main trial We 
assessed feasibility through progress in the pilot trial 
rather than separately. In hindsight, qualitative methods 
would have been a useful addition to allow exploration 
of clear difficulties that subsequently arose. 

Four centers were involved in recruiting pilot trial 
women who were new outpatient attendees at gynecology 
clinics with symptomatic prolapse and fitted with a pes- 
sary. Women were eligible to participate in this study if 
they were 18 years or older, confirmed to have prolapse of 
any type, of severity stage I to IV (as measured on vaginal 
examination using the pelvic organ prolapse quantitation 
(POP-Q) method, [27]), and were successfully fitted with a 
pessary which was still in place after 2 weeks. Women 
were excluded if they had previously had a pessary or been 
formally taught PFMT, if there were contraindications to 
either of these treatments, or if they were unable to give 
informed consent. The women targeted for the pilot trial 
had been screened for another trial, for which they had 
been ineligible due to the fact they had a pessary fitted. A 
patient information leaflet was provided and written 
informed consent obtained. Ethical approval for the study 
was obtained from the West of Scotland Research Ethics 
Committee (REC) 1 on 7 December 2007 (07/S0703/141). 

At the appointment when the pessary was fitted, 
women were told about the study. If agreeable, women 
were then contacted by the study office to discuss their 
participation. Consenting participants were randomized 
into one of two groups: 1) PFMT (delivered by a special- 
ist women's health physiotherapist at five appointments 
over 16 weeks) in conjunction with pessary management 
of their prolapse (intervention group); or 2) pessary 
management alone (control group). Participants had a 
nurse appointment 6 months after randomization at 
which time their pessary was removed for 1 month, and 
an assessment of vaginal tissues and symptoms was 



undertaken. At 7 months after randomization, participants 
attended a review appointment with their gynecologist to 
have their prolapse re-assessed, the pessary re-fitted if 
necessary, and other treatment needs discussed. Parti- 
cipants received postal questionnaires at baseline (after 
pessary fitted but prior to randomization), 6 months post- 
randomization (prior to seeing the nurse to have their 
pessary removed), and 7 months post-randomization 
(prior to the 7-month gynecologist review appointment), 
and recorded symptoms in a diary for 1 month after re- 
moval of the pessary at 6 months. The primary outcome of 
the pilot trial was prolapse symptoms measured using the 
pelvic organ prolapse symptom score (POP-SS) [26]. Other 
secondary prolapse outcomes were: prolapse-related quality 
of life, prolapse severity (POP-Q) [27], and perceived 
change in prolapse since pessary fitted. POP-Q assessment 
at 7 months was undertaken blind to the group allocation 
and previous POP-Q results. 

Analysis methods 

No systematic guidance on how to categorize and ex- 
plore issues that have arisen in a feasibflity study was 
identified. However, Shanyinde et al [15] reported 14 
issues that need to be evaluated in feasibflity studies or 
pflot trials. Shanyinde et al, were not completely clear 
on how those 14 issues were developed; however, some 
issues may have arisen from the detafled literature review 
they had undertaken, and afl appear to have a degree of 
face validity. We used the list of 14 methodological issues 
(as the best available) to categorize and assess the extent 
to which each issue was addressed (or not) in this feasibfl- 
ity study with pflot trial. Subsequently, we developed a 
process to support robust and systematic decision-making 
in moving from feasibflity study to fufl trial and on to real- 
world implementation. The feasibility and pflot data from 
the exemplar study were examined in order to address 
issues highlighted by using the Shanyinde et al methodo- 
logical issues as a framework [15], and to test out the 
newly-developed process. Pflot trial data were analyzed 
using descriptive statistics, but no statistical testing was 
undertaken as the study did not aim to draw inferences 
from the data. 

Results 

The results of applying the methodological issues as an 
analytic framework [15] to the exemplar study findings 
are presented under each of the 14 items summarized in 
Table 1. 

Sample size calculation 

The number of participants recruited in the pilot trial 
was smaU (n = 16 out of the target of 50 participants); 
indeed, poor recruitment was a key issue within the 
study. The sample size for a potential main trial was. 
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Table 1 Summary of findings against 14 methodological issues for feasibility research 


Methodological issues 


Findings 


Evidence 


1. Did the feasibility/pilot study 
allow a sample size calculation 
for the main trial? 


Achieved even though sample 
was small 


16 out of target of 50 participants achieved 
in feasibility study 

103 per group indicated for main trial 


2. What factors influenced eligibility 
and what proportion of those 
approached were eligible? 


Ineligibility for randomization was 
mainly due to participant refusal 


31 out of 66 approached were eligible 


3. Was recruitment successful? 


Recruitment was very difficult. Issues 
at center, clinician, and participant levels 


Centers showed low enthusiasm. 

Clinicians failed to identify participants 

Eligible participants not willing to take part 
(11/31 withdrew pre-randomization) 


4. Did eligible participants consent? 


Low conversion to consent 


16 (52%) randomized out of 31 eligible 
participants 


5. Were participants successfully 
randomized and did randomization 
yield equality in groups? 


Worked well 


Equal sized groups, well-balanced on the 
minimization variables 


6. Were blinding procedures 
adequate? 


Where used, blinding worked well 


Self-reported evidence from gynecologists 
suggests blinding of POPQ measurement 
was successful 


7. Did participants adhere to the 
intervention? 


Good adherence to PFMT appointments 
and less so to diaries 


Physiotherapy appointments attended: n = 5, 
62.5%; n = 4, 1 2.5%; n = 3, 1 2.5%; n = 2, 1 2.5% 

Exercise diaries returned: n = 4, 50%; 
n = 3, 25%; n = 2, 12.5%; n = 0, 12.5% 


8. Was the intervention acceptable 
to the participants? 


Not directly assessed but low numbers 
recruited suggest some difficulty 


15 eligible participants decided not to be 
randomized once all information was available 


9. Was it possible to calculate 
intervention costs and duration? 


Assessed in the linked trial 


Cost of PFMT: £1 70 for mean of 4.2 
appointments attended 


10. Were outcome assessments 
completed? 


Outcome measures used did 
assess main areas of interest 


See summary of outcome data in Table 2 


11. Were outcomes measured those 
that were the most appropriate 
outcomes? 


Sexual problems questionnaire 
completion poor 


Only 25% of participants completed all 
items at baseline 


12. Was retention to the study 
good? 


Once recruited retention was good 


Response rates: 

6- month questionnaire: 100% 1, 87.5% C 

7- month questionnaire: 75% 1, 37.5% C 
Symptom diary completion: 62.5% 1, 37.5% C 

6- month appointment: 87.5% 1, 50% C 

7- month appointment: 75% 1, 37.5% C 
Agreed to pessary removal: 66.7% 1, 33.3% C 


13. Were the logistics of running 
a multicenter trial assessed? 


Some centers recruited better than 
others. The center that recruited well 
had a dedicated local recruiter 


Center 1:15 participants (dedicated recruiter) 
Center 2: 1 participant 
Center 3: 0 participants 
Center 4: 0 participants 


14. Did all components of the 
protocol work together? 


Components had strong synergy 


There were no difficulties identified in the 
various processes and the researcher's ability 
to implement them. For example, if participants 
were recruited, they were easily randomized 
and their care moved forwards to the 
appropriate trial arm. 



Methodological Issues based on Shanylnde ef ai [15]. C, control; I, intervention; PFMT, pelvic floor muscle training; POP-Q, pelvic organ prolapse quantitation. 
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however, calculated based on the difference between 
groups in the primary outcome measure observed at 6 
months. It was estimated that a future trial powered to 
detect the observed difference (3 points on the POP-SS, 
SD 7) would require 103 participants per group (90% 
power, 5% level of significance, two-sided). Our work on 
developing the POP-SS instrument, which is based on 
women undergoing surgery or PFMT, has given a pre- 
liminary estimate of the minimally important change in 
POP-SS score of 1.5 [28]. Therefore, as women with 
pessary treatment may be different from those on whom 
original testing was undertaken, we have chosen a 
conservative estimate POP-SS score of 3 between groups 
to ensure the difference is meaningful. Given the small 
sample size indicated, the question remains: are the 
observed difference between groups and SD good indica- 
tors on which to base such calculations? Our other stud- 
ies using the POP-SS suggest that the SD differs between 
populations: treatment naive participants with prolapse 
in a trial of PFMT, SD 6; participants after surgery for 
prolapse, SD 8; and participants without prolapse in a 
longitudinal study of symptoms after childbirth, SD 3.5. 
This would suggest that the SD observed in this feasibil- 
ity study was in keeping with that in other groups of 
participants known to have a prolapse. If the SD was 
artificially inflated the impact on sample size calculations 
would be to increase the sample size, thus providing an 
overestimate of what was needed. Establishing the differ- 
ence between groups in the primary outcome measure is 
of course the aim of the main trial and cannot be accur- 
ately estimated in a small sample. Examining estimates 
of effect size from other studies which have measured 
the same outcome can also help to inform the main trial 
sample size. 

Eligibility 

During the period from 1 February 2008 to 11 February 
2010, a total of 66 women were approached to take part 
(Figure 1, Consolidated Standards of Reporting Trials 
(CONSORT) diagram). Of these, 24 women were ineli- 
gible mainly due to being unwilling to be contacted by 
the study team after being provided initial information 
by clinical staff (n = 20). Of the 42 eligible women, 11 
did not successfully retain their pessary so became ineli- 
gible. A further 15 women either decided not to partici- 
pate when the study office contacted them to obtain 
consent, or withdrew once they received paperwork 
prior to randomization. The remaining 16 participants 
were randomized, eight to intervention and eight to the 
control group. 

Recruitment 

Recruiting to the study proved very difficult, despite an 
extended recruitment period and the addition of an 



extra center. Three centers effectively failed to initiate 
recruitment. Reasons given by those centers included 
extremely busy clinics and limited use of pessaries by 
the recruiting gynecologists. In addition, many women 
were reluctant to participate as noted above. As women 
were recruited at a point when the decision to fit a pes- 
sary had already been made, this may have affected their 
motivation to agree to being randomized to receive an 
additional intervention. 

For a considerable number of women, not taking part 
in the study seemed linked to the fact that they saw the 
pessary as a permanent solution; if the pessary relieved 
their symptoms they would keep it in the long-term and 
there would be no need for additional intervention. One 
of the hypotheses, however, was that the combination 
of PFMT and pessary might allow women to become 
symptom-free, or have reduced symptoms, without the 
need for a pessary. Thus, to be recruited, women needed 
to view the pessary as potentially a short-term compo- 
nent of their treatment. The fact that some women did 
not wish to have their pessary removed at 6 months 
indicates that this was not always the case. 

There was an indication that participants who entered 
the study were younger and had less severe prolapse 
than those who did not. This perhaps suggests that the 
older participants with more severe prolapse favored a 
pessary on a more permanent basis and therefore were 
less inclined to take part. 

Consent 

Of the 66 women approached by their gynecologist to 
take part, 20 were unwilling at the outset to be con- 
tacted by the study team, and prior to randomization a 
further 15 women decided not to participate. Thus, over 
half of the women approached and otherwise eligible 
chose not to consent to take part. Better counseling of 
women by clinical staff at study centers may have 
avoided this problem. In the center where there was a 
dedicated person undertaking screening, gaining consent 
was more successful. 

Randomization procedures 

Once participants had completed and returned their 
consent form their group allocation was determined. 
The research assistant randomized participants into one 
of the two groups using a randomization programme in 
Access (Microsoft, Redmond, WA, USA) developed by 
the Centre for Healthcare Randomized Trials (CHaRT), 
University of Aberdeen, Aberdeen, UK. Randomization 
was minimized on center, stage of prolapse (I, II, III, IV), 
type of pessary fitted (ring or shelf), and whether topical 
estrogen was being used (yes, no). The processes for 
achieving randomization worked smoothly. 
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66 Women Approached 



Number ineligible n=24 



Pre-Randomisation 
Withdrawal: 

Pessary fell out n=1 1 
Changed mind n=15 



Number eligible 
n=42 



Number randomised n=16 




Intervention 
n=8 



Withdrew n=1 



Numbers for analysis, intervention 
group: 

• Number returning 6 month 
Questionnaire n=8 

• Number attending 6 month nurse 
appointment n=7 

• Number agreeing to have pessary 
removed for 4 week period n=6 

• Number returning symptom diary 
n=5 

• Number returning 7 month 
questionnaire n=6 

• Number attending 7 month review 
with gynaecologist n=6 



Figure 1 CONSORT diagram. 



Reasons for ineligibility*: 
Unwilling to be contacted n=20 
On waiting list for surgery n=6 
Preference for PFMT n=4 
Previous surgery n=2 
Previous PFMT n=2 
Other n=7 

*women may have multiple reasons for 
withdrawal/ineligibility 



Control 
n=8 



Withdrew n=4 



Numbers for analysis, control group: 

• Number returning 6 month 
Questionnaire n= 7 

• Number attending 6 month nurse 
appointment n=4 

• Number agreeing to have pessary 
removed for 4 week period n=3 

• Number returning symptom diary 
n=3 

• Number returning 7 month 
questionnaire n=3 

• Number attending 7 month review 
with gynaecologist n=3 



Blinding procedures 

Blinding of participants and the intervention physiother- 
apists was not possible given the nature of the interven- 
tion. However outcome assessment was mainly through 
participant-completed questionnaires, thus avoiding as- 
sessor bias. Gynecologists were blinded to participants' 
group allocation at the 7-month follow-up prolapse re- 
view appointment. 

Adherence to intervention 

Adherence in terms of replying to study questionnaires 
was high at baseline and 6 months, but at 7 months was 
poorer, particularly in the control group. Attendance at the 
6-month nurse appointment and the 7-month gynecology 
review appointment was poor in the control group. 



Adherence with attending physiotherapy appointments 
was good, with 75% of participants in the intervention 
group attending four or five of the planned five appoint- 
ments, and all except one participant completed the exer- 
cise diaries between appointments. 

Acceptability of intervention 

Acceptability was not assessed directly but adherence to 
physiotherapy gives an indication, as does the initial 
(poor) rates of willingness to participate. It may be that 
the intervention and the trial processes were acceptable 
to some participants (those that participated and ad- 
hered to the protocols) but not others (those who chose 
not to take part, perhaps due to being put off by the 
intervention or the associated trial processes, or by 
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other commitments). Post-randomization withdrawals 
were common in the control group suggesting that con- 
trol participants may have been dissatisfied with their 
allocation. 

Cost and duration of intervention 

No economic evaluation was included in the study but 
cost information was available from a related trial about 
cost of PFMT [29]. 

Outcome assessment 

The participants mainly had prolapse involving the anter- 
ior vaginal wall, of stage II or III severity. Of the 16 partici- 
pants randomized, five withdrew post-randomization (one 
intervention and four control participants) for various rea- 
sons; however, four of these participants continued to 
complete questionnaires. Two participants did not wish 
their pessary to be removed at the 6-month stage. Seven- 
month POP-Q data were available for nine participants. 

The randomized groups were comparable at baseline 
with regard to demographic and outcome variables. The 
mean age of participants who were not randomized was 
slightly older (70.52 years, SD 13.56, n = 40) compared 
to those randomized (63.12 years, SD 14.301, n = 16) 
suggesting a slight selection bias towards younger 
women participating in the study (t = 1.817, df = 54, P = 
0.075). Non-randomized participants also tended to have 
more severe prolapse (stage I, 0%; stage II, 60%; stage 
III/IV, 40%) compared to randomized participants (chi- 
squared test = 10.920, df = 2, P = 0.004). 

The mean POP-SS score was greatest (more frequent 
symptoms) in both groups at the 7-month time-point, 
indicating worse symptoms after the pessary had been 
removed (Table 2). The mean score was higher in the 
control group compared to the intervention group at 6 
months, but the reverse was true at 7 months. 

Comparable changes in POP-Q stage were observed in 
the intervention and control group (Table 2). There was 
no difference between the study groups in terms of the 
subjective change in prolapse that participants reported 
in the 6- and 7-month questionnaires (Table 2). 

Selection of most appropriate outcomes 

The most appropriate outcomes to use were decided be- 
forehand based on the experience of the researchers in 
other prolapse studies [26,29-31]. However, response to 
the sexual symptom questionnaire was poor with some 
participants finding these questions intrusive. 

Retention 

Information on follow-up questionnaire rates is included 
under the adherence section. Once participants were 
randomized follow-up was generally good, both in terms 
of attendance at appointments and completion of 



Table 2 Summary of primary outcome (POP-SS) and main 
secondary outcomes at baseline, 6 months, and 7 months 



POP-SS score (scores presented Intervention Control 

as mean (SD) median, range, n)^ 



col i no 


^.J / \J).0/ ) 


Q nn (1 ic\\ 

J.UU \A.Z.\J) 
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n — R 
n — O 


n — 7 
n — / 


7 months 


15.33 (6.35) 


12.67 (9.24) 




19, 8 to 19 


18, 2 to 18 




n = 3 


n = 3 


Change in POP-Q stage^ 


n = 6 


n = 3 


+ 2 stages 


0 


0 


+ 1 stage 


2 (33%) 


2 (66%) 


No change in stage 


3 (50%) 


1 (33%) 


-1 stage 


0 


0 


-2 stages 


1 (17%) 


0 


How do you feel your prolapse 
is now compared to when your 
pessary was first inserted? 






6 months 


n = 7 


n = 7 


Better 


5 


5 


Same 


2 


1 


Worse 


0 


1 


7 months 


n = 6 


n = 3 


Better 


3 


1 


Same 


2 


1 


Worse 


1 


1 



^POP-SS score: 0 = none, 28 = all symptoms all the time; "^a negative value 
indicates an improvement at 6 months. POP-SS, pelvic organ prolapse 
symptom score; POP-Q, pelvic organ prolapse quantitation. 



questionnaires, suggesting that this part of the method- 
ology would be transferrable to a larger trial. Adherence 
did appear to be poorer in the control group, and com- 
ments from some participants suggested that this was 
because they considered they were missing out' due to 
their allocation to the control group. 

Logistics of multicenter trial 

Some centers failed to recruit, suggesting that there may 
be some difficulty with implementing a multicenter trial. 
The center that recruited the majority of participants 
had dedicated local researchers responsible for local trial 
operationalization and recruitment. 

All components of the protocol work together 

Undoubtedly, we gained experience of delivering the 
various components contained within a protocol that 
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aimed to deliver and evaluate a complex intervention. 
This was the main strength of running the trial on a 
small scale beforehand, which was a miniature version of 
the (then) planned main trial 

Discussion 

Despite some aspects of the pilot study running relatively 
smoothly (for example, randomization and general adher- 
ence), data revealed the existence of a number of feasibility 
issues in designing and operationalizing a full trial These 
included problematic center/ clinician engagement, low 
initial patient willingness to participate, significant drop 
outs post-consent but pre-randomization, introduction of 
potential selection bias at recruitment, and low levels of 
data completion with regard to exercise diaries. 

Methodological choices, challenges, and trade-offs in 
responding to the feasibility results 

Given the issues outlined, we concluded that the trial 
was not feasible in its current form. The study did sug- 
gest that issues such as randomization, blinding, adher- 
ence to the intervention, outcome assessment, retention, 
and all components working together were reasonably 
non-problematic. In addition, sufficient data was ob- 
tained to inform a sample size calculation. However, a 
number of important changes would also need to be 
made: we lacked information about the population to 
whom this intervention may be best targeted towards, 
mainly in terms of acceptability; despite some helpful 
studies of pessary use [19,32,33] we lacked information 
about use of pessaries by clinicians (recruitment); we 
identified that recruiting post-pessary treatment decision 
was probably not the best strategy (eligibility/recruit- 
ment/consent); we identified the need to change to using 
dedicated local recruiters (recruitment/consent); con- 
temporary evidence published after the start of the feasi- 
bility study also suggested a third arm to the trial should 
be added (intervention); and one outcome was poorly 
completed and needed reconsideration (outcomes). 

Consideration of the challenges highlighted by the feasi- 
bility study revealed two principle methodological chal- 
lenges: 1) deciding what to change in the light of the pilot 
study findings (the intervention, the trial design, and/or 
the clinical context) and the implications of those changes 
for the explanatory or pragmatic nature of the trial; and 2) 
a lack of formal or informal guidance on how decision- 
making in response to pilot trial findings should be made 
and/or documented. These are explored in turn below. 

How should decisions regarding what to change as a 
result of the findings of a pilot RCT be made? 

We identified four potential options in regard to ad- 
dressing the problems identified: 1) adapting the inter- 
vention; 2) adjusting the clinical context within which 



the intervention would be delivered; 3) amending ele- 
ments of the trial design; or 4) a combination of any of 
the former (Figure 2). We soon found that many of the 
problems identified in the feasibility study could be ad- 
dressed by targeting any or all of the above. However, 
further consideration revealed that some solutions may 
solve the problems for the trial, but would not work sub- 
sequently in the real-world', thereby storing up the prob- 
lem for later implementation. Below we demonstrate 
this in regard to recruitment. 

If we assume that problems faced around recruitment 
of participants stemmed largely from a lack of accept- 
ability of the intervention, then a challenge is imme- 
diately faced. Given the fundamental nature of the 
intervention, including its relative simplicity, there were 
few parameters that are open to adaptation. One might 
be the use of financial incentives to aid recruitment and 
even adherence. Certainly, this is gaining increasing 
popularity in regard to a range of health and illness 
behaviours [34,35] and is showing some degree of effect- 
iveness, at least within the short-term [36,37]. The deci- 
sion to use incentives is methodologically acceptable and 
compatible with an explanatory trial. Demonstrating 
potential benefit is central to such trials and as such ne- 
cessitates strong study control and maximizing high 
patient adherence [13,38]. Indeed, Thorpe et al argued 
that 'the more rigorous a trial is in measuring and 
responding to noncompliance of the study participants, 
the more explanatory it becomes' [39], p.54. However, 
we had originally intended the purpose of the final trial 
to be largely pragmatic, demonstrating potential real- 
world benefit and thereby providing helpful information 
for both policy makers and clinicians in their decision- 
making processes [40]. In order to achieve this, the 
benefit and feasibility of the intervention must be dem- 
onstrated in real-world conditions. With regard to ad- 
herence, Godwin et al [12] suggested that this means 
that attempts can be made to improve compliance, but 
these must not go beyond what would be expected in 
routine practice. We are therefore left with two apparent 
choices: to use incentives to increase recruitment and 
adherence but provide results from a final trial that are 
inapplicable to the real-world, or not to use incentives 
and to conduct a pragmatic trial that is likely to show 
effectiveness despite potential benefits (if only partici- 
pants would adhere). 

Despite this apparent tension, we would argue a third 
option exists: to redefine incentives as an added parameter 
of the intervention rather than representing a change to 
context/routine clinical practice. This means increasing 
the complexity of the intervention based on empirical data 
within the feasibility study; the intervention now has two 
components: pessary and incentive. If defined as such then 
the benefits of external validity (generalizability), the 
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PROBLEM TYPE 



TYPE A: Issue likely to 
be a problem ONLY FOR 
THE TRIAL (i.e. unlikely 
to be a problem in the 
real world) 

EVIDENCE: 



TYPE B : Issue is likely to 
be a problem for BOTH 
THE TRIAL AND THE 
REAL WORLD 

EVIDENCE: J 



TYPE C: Issue is likely to 

be a problem 

ONLY FOR THE REAL 

WORLD 

EVIDENCE: ^ 



SOLUTIONS 



CHANGE ASPECTS OF: 




a) INTERVENTION 

1) 
2) 

3) 

b) TRIAL DESIGN 
1) 

W 



I 



ASSESSMENT OF SOLUTIONS 

Could solution be effective in trial setting? 

EVIDENCE: 

_A L 



yes 



I 




Could solution be feasible in trial setting? 

EVIDENCE: 




1 



yes 




CHANGE ASPECTS OF: 



a) INTERVENTION 

1) 

IP 

b) TRIAL DESIGN 

1) 
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Could solution be effective in real world? 

EVIDENCE: 




c) CONTEXT 



1) 
2) 
3) 



yes 



|Could solution be feasible in real world? 



no 



yes 



1 



EVALUATION OF SOLUTIONS 

Box 1 : Trial Context 
Stage 1 : Rank options by likely 
degree of effectiveness and 
feasibility- in the trial. 
Stage 2 : Assess potential to combine 
solutions considering any likely 
interaction in terms of feasibility and 
effectiveness- in the trial. 
Stage 3 : Assess the most cost- 
effective single or multiple solution - 
in the trial. 

In order to maximise external validity 
of the trial prioritise options which 
achieve highly on Box 1 AND Box 2. 
IF there is a trade-off in effectiveness 
for any solution(s) between Box 1 and 
Box 2 then: 

• For a more explanatory trial 
prioritise Box 1 

• For a more pragmatic trial a 
prioritise Box 2 



Box 2 Real World Context 
Stage 1 : Rankoptions by likely 
I degree of effectiveness and 



1 



feasibility- in the real world. 

Stage 2 : Assess potential to combine 
solutions considering any likely 
interaction in terms of feasibility and 
effectiveness- in the real world. 
Stage 3 : Assess the most cost- 
effective single or multiple solution 
in the real world 



i 



Figure 2 A process for Decision-making after Pilot and feasibility Trials (ADePT). 



strength of the pragmatic design, is retained as the infer- 
ence from the trial and that benefit stems from pessary 
use plus incentives within routine clinical practice rather 
than pessary alone. The inclusion of additional interven- 
tion parameters in order to address contextual weaknesses 
is not new, and may more commonly be included as the 
educational component required for clinicians to fully 
engage with and deliver the intervention [41]. If the added 
parameter is financial, it may be beneficial to consider eco- 
nomic evaluation within feasibility work, particularly in 
the context of pragmatic trials. 

Pragmatic trials are known to frequently test interven- 
tion strategies' rather than simple devices per se, allow- 
ing greater clinician and patient discretion with regard 
to the adoption and use of the intervention [38]. How- 
ever, the current literature on explanatory versus prag- 
matic trials has not adequately addressed the question as 
to how or when such a strategy is developed or decided 
upon, nor has the literature on feasibility and pilot stud- 
ies. The 2000 version of the MRC framework for com- 
plex interventions suggests a linear process in which 



feasibility and pilot testing are conducted after a period 
of modeling and intervention definition, thus the inter- 
vention is largely defined [42,43]. This study, however, 
suggests that the process of intervention development 
and definition may well continue post-feasibility and 
pilot. This is certainly more akin to the less linear and 
more iterative process included in the updated 2008 
framework [1]. 

Can guidance to support evidence-based and 
documented decision-making in response to pilot RCT 
findings be produced? 

The final methodological problem that we faced was the 
lack of any detailed guidance to support the complex 
decision-making required, and there did not appear to be 
any requirement to make the alternative amendments, 
their consequences, or the rationale for the final choice 
explicit to external audiences. At the heart of any such 
guidance must be the assurance that the endpoint will be 
a trial design that is fit for purpose [13]. 
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Ultimately, we would argue that by the introduction of 
greater rigor at this stage of the intervention and trial 
design process, the final utility, through real-world ef- 
fectiveness and likelihood of real-world implementation, 
will be significantly enhanced. Several authors have 
highlighted that funders have historically preferred the 
rigor and internal validity offered by explanatory trials 
[44]. Certainly, this is the case for pharmaceutical com- 
panies where the explanatory trials are likely to show 
larger effect sizes for more narrowly defined patients 
within more controlled circumstances. However, the 
move towards more pragmatic trials has been largely 
fuelled by an increasing willingness to apply RCT 
methods to more complex interventions (which natur- 
ally tend to be less well-defined [45], more flexible [46], 
and more context-dependent [45]), and a recognized 
need to provide information of greater utility to policy- 
makers, clinicians, and patients [40,47,48]. If utility to 
stakeholders is a likely criterion of success then it would 
seem that their perspectives would need to be taken into 
account when assessing the varied strategies that might 
be adopted in response to problems highlighted by feasi- 
bility studies. Again, this suggests not only that current 
guidance on criteria that feasibility and pilot studies 
should address might benefit from expansion, but also 
the process of suggestions and recommendations about 
decision-making processes as well. It is this gap that A 
process for Decision-making after Pilot and feasibility 
Trials (ADePT) seeks to fill. 

ADePT 

Given the tensions highlighted, we sought to establish a 
systematic means of supporting and documenting future 
decision-making in this area. Consequently, we devel- 
oped the ADePT process to: 1) encourage the systematic 
identification and appraisal of problems and potential so- 
lutions; 2) improve the transparency of decision-making 
processes; and 3) reveal the tensions that exist between 
pragmatic and explanatory choices. The process is shown 
in Figure 2, with a worked example addressing the issue 
of acceptability of the intervention to the PEPPY study 
shown in Additional file 1. The ADePT process includes 
3 key steps. 

Step 7: Deciding on the type of problem experienced and 
the evidence to support that allocation 

Three types of problems are possible: type A, the issue is 
likely to be a problem only for the trial (that is, unlikely to 
be a problem in the real-world); type B, the issue is likely 
to be a problem for both the trial and the real-world; and 
type C, the issue is likely to be a problem only for the 
real-world. 

For each type, it is necessary to expose the evidence 
for why the problem exists and what type of problem it 



is. In the example (Additional file 1), evidence from the 
failure to achieve recruitment and from methodological 
papers suggest that acceptability of the intervention (the 
pessary) was a problem. Furthermore, the intervention 
(pessary) is an existent clinical tool and not amendable 
to adaptation; therefore, the acceptability issue was 
highly likely to be a problem for the trial and also a 
problem for the real-world implementation. This typ- 
ology is crucial for later decision-making in order to 
facilitate the choice of a solution that will not only pro- 
vide a short-term solution for the trial but also, if neces- 
sary, a longer-term solution that would be possible to 
implement in the real-world. 

Step 2: Identifying the range of potential solutions and the 
evidence to support those solutions 

Potential solutions may apply to the intervention, the 
trial design, or the cUnical context. Solutions can be iden- 
tified in various ways, including: literature searching; de- 
bate/ brainstorming within the research team (ideally 
including lay representation); and, if available, through 
analysis of feasibility study data. The documentation of 
the identified solutions lays the process open to scrutiny, 
and hence debate and transparency. Depending on the 
type of problem (type A, B, or C), each solution will need 
to be assessed in regard to its ability to address or ameU- 
orate the problem in the trial, or real-world clinical con- 
text, or both. The assessment of each option should be in 
terms of: 1) the potential effectiveness of the solution; 
and 2) the potential feasibility of the solution. Evidence to 
support each assessment should, ideally, be provided. 
In the documented example (Additional file 1) we 
assessed solutions through searching the research lite- 
rature (focusing on methodological reports and ex- 
periences reported within cUnical trials), and focused 
discussions with colleagues who had experience of simi- 
lar issues and/or potential solutions. 

Step 3: The assessment of best options 

Once all options have been appraised then a final deci- 
sion should be made. For type A and type C solutions, a 
simple ranking and assessment as to whether a single or 
combined solution might be best is likely to suffice; this 
may also incorporate an assessment of associated costs. 
For type B problems, a final assessment is likely to be 
more complex. It is possible that a solution may be iden- 
tified that is judged to work extremely well in both the 
trial and real-world, in which case this should be chosen 
as this option will effectively minimize the tension be- 
tween internal and external validity - the explanatory 
trial becomes increasingly synonymous with the prag- 
matic trial. However, we anticipate that in the majority of 
cases a judgment may have been made that some solu- 
tions may work extremely well within the trial but will 
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work less well within the real- world (or vice versa). In this 
situation the research team and others may be forced to 
choose where on the continuum of explanatory to prag- 
matic [39] they wish to place the trial, and justify this. 

We would acknowledge that obtaining robust evidence 
to support choices and trade-offs at each of these stages 
is likely to prove difficult. However, decisions will have 
to be made even if only on tentative evidence or argu- 
ment. In other areas facing this dilemma, for example, 
the creation of clinical guidelines, forms of consensus 
method have been effectively employed [49]. It is pos- 
sible that this could be explicitly adopted to improve the 
transparency and rigor of stakeholder decision makers 
within feasibility studies. Indeed, while Godwin [12] and 
others have highlighted the frequent trade-off between 
internal and external validity, it is possible that the in- 
clusion of wider stakeholders in the generation and as- 
sessment of solutions may more effectively identify ways 
of achieving both, or at least minimizing trade-off, for 
any particular intervention and trial. Documenting and 
exposing the decision-making process becomes even 
more important when the evidence to support decision- 
making is weak, because laying the process bare pro- 
motes transparency, robustness, and exposure of the 
trade-offs made between a more explanatory and a more 
pragmatic design. 

Conclusions 

This feasibility and pilot study successfully gained know- 
ledge about the process and implementation of the pro- 
posed main trial. It has also identified the gaps in our 
knowledge that require to be filled prior to progressing 
to a main trial. We have undertaken a further survey of 
professional practice to supplement the study [50] and 
plan a small qualitative study to explore acceptability to 
participants. Although the trial is not feasible in the 
tested form, some adaptations and the further studies 
will allow us to progress forwards to a stronger main 
trial. We advocate the use of Shanyinde et aUs work to 
structure discussion about what has been, or could be 
learned, from a feasibility study and where the gaps 
remain [15], and testing the ADePT process to support 
systematic and transparent identification of issues and 
solutions for the design of future trials. Our on-going 
work with ADePT will explore its usefulness when 
developing an intervention and designing the trial to test 
it. The worked example and the identification of trade- 
offs between previously unacknowledged options have 
raised the question as to whether a pilot RCT should be 
conducted to assess the feasibility of an explanatory/ 
pragmatic trial, or to decide whether it should be ex- 
planatory or pragmatic. Further consideration of this 
issue is recommended. 



Additional file 



Additional file 1: Worked example using ADePT [51-69]. 
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